feat: filter out docs #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft

forshev wants to merge 23 commits into main from 290-filter-out-docs

Contributor

forshev commented Dec 12, 2025 •

edited

Loading

Description

This is PoC of "delete imitation"

Fixes #290

I have read and followed all requirements in CONTRIBUTING.md;
I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

forshev marked this pull request as draft

December 12, 2025 13:33

codecov-commenter commented Dec 12, 2025 •

edited

Loading

Codecov Report

❌ Patch coverage is 53.66748% with 379 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.63%. Comparing base (6ae5683) to head (e39db32).

Files with missing lines	Patch %	Lines
docsfilter/docs_filter.go	22.83%	159 Missing and 10 partials ⚠️
docsfilter/loader.go	44.44%	19 Missing and 16 partials ⚠️
docsfilter/encoding.go	78.80%	17 Missing and 15 partials ⚠️
util/fs.go	38.77%	17 Missing and 13 partials ⚠️
frac/active_index.go	63.04%	15 Missing and 2 partials ⚠️
docsfilter/filter.go	0.00%	16 Missing ⚠️
cmd/seq-db/seq-db.go	0.00%	14 Missing ⚠️
docsfilter/merged_iterator.go	81.48%	7 Missing and 3 partials ⚠️
frac/sealed_index.go	70.96%	8 Missing and 1 partial ⚠️
docsfilter/iterator_asc.go	75.00%	5 Missing and 3 partials ⚠️
... and 8 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #297      +/-   ##
==========================================
- Coverage   71.49%   70.63%   -0.86%     
==========================================
  Files         204      213       +9     
  Lines       14857    15609     +752     
==========================================
+ Hits        10622    11026     +404     
- Misses       3471     3764     +293     
- Partials      764      819      +55

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cheb0 reviewed

View reviewed changes

docsfilter/metrics.go Outdated

+              	diskUsage = promauto.NewGaugeVec(prometheus.GaugeOpts{
+              		Namespace: "seq_db_store",
+              		Subsystem: "filters",
+              		Name:      "disk_usage_bytes_total",

Member

cheb0 Dec 15, 2025

According to company standards, only counters now can (and must) have _total suffix. Gauges can not end with _total

forshev force-pushed the 290-filter-out-docs branch from 6b1e6c5 to f98406a Compare

December 26, 2025 11:24

forshev force-pushed the 290-filter-out-docs branch from 35c810c to 7baeaef Compare

January 21, 2026 09:05

Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/medium-4`	`467eb3`	`61c22e`
	`9583.66 MB/s`	`7535.90 MB/s`	`0.79`	`🔴`
	`106.80 ns/op`	`135.90 ns/op`	`1.27`	`🔴`
`FindSequence_Random/small-4`	`467eb3`	`61c22e`
	`5128.78 MB/s`	`4585.96 MB/s`	`0.89`	`🔴`
	`49.91 ns/op`	`55.82 ns/op`	`1.12`	`🔴`
`Indexer-4`	`467eb3`	`61c22e`
	`265877562.00 ns/op`	`296289980.00 ns/op`	`1.11`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 7baeaef to 5e040ab Compare

January 21, 2026 09:10

Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`467eb3`	`1045a0`
	`5128.78 MB/s`	`3914.73 MB/s`	`0.76`	`🔴`
	`49.91 ns/op`	`65.39 ns/op`	`1.31`	`🔴`
`GenerateDocs-4`	`467eb3`	`1045a0`
	`975.70 ns/op`	`1179.00 ns/op`	`1.21`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 5e040ab to 5fb5af0 Compare

January 21, 2026 12:43

Contributor

github-actions bot commented Jan 21, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`ESBulk-4`	`abfd3f`	`5035be`
	`378.00 B/op`	`444.00 B/op`	`1.17`	`🔴`
`FindSequence_Random/medium-4`	`abfd3f`	`5035be`
	`12756.46 MB/s`	`9068.94 MB/s`	`0.71`	`🔴`
	`80.27 ns/op`	`112.90 ns/op`	`1.41`	`🔴`
`FindSequence_Random/small-4`	`abfd3f`	`5035be`
	`6511.28 MB/s`	`4151.19 MB/s`	`0.64`	`🔴`
	`39.32 ns/op`	`61.67 ns/op`	`1.57`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 5fb5af0 to 277508f Compare

January 23, 2026 09:59

Contributor

github-actions bot commented Jan 23, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Bitmask-4`	`32489f`	`90d653`
	`0.00 B/op`	`3.00 B/op`	`NaN`	`🔴`
`FindSequence_Random/medium-4`	`32489f`	`90d653`
	`9785.69 MB/s`	`8062.37 MB/s`	`0.82`	`🔴`
	`104.60 ns/op`	`127.00 ns/op`	`1.21`	`🔴`
`GenerateDocs-4`	`32489f`	`90d653`
	`958.10 ns/op`	`1143.00 ns/op`	`1.19`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 4846a3e to 3deb507 Compare

February 3, 2026 11:10

Contributor

github-actions bot commented Feb 3, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`174405`	`7d5d8f`
	`10280.15 MB/s`	`4422.80 MB/s`	`0.43`	`🔴`
	`24.90 ns/op`	`57.88 ns/op`	`2.32`	`🔴`
`FindSequence_Random/tiny-4`	`174405`	`7d5d8f`
	`2491.86 MB/s`	`2183.17 MB/s`	`0.88`	`🔴`
	`25.68 ns/op`	`29.32 ns/op`	`1.14`	`🔴`
`GenerateDocs-4`	`174405`	`7d5d8f`
	`952.20 ns/op`	`1126.00 ns/op`	`1.18`	`🔴`
`ProcessDocuments-4`	`174405`	`7d5d8f`
	`381.00 B/op`	`33066.00 B/op`	`86.79`	`🔴`
	`4.00 allocs/op`	`5.00 allocs/op`	`1.25`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 3deb507 to 6b2cfb2 Compare

February 3, 2026 14:03

Contributor

github-actions bot commented Feb 3, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/small-4`	`174405`	`4712c5`
	`10280.15 MB/s`	`4097.76 MB/s`	`0.40`	`🔴`
	`24.90 ns/op`	`62.47 ns/op`	`2.51`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 6b2cfb2 to 37173f0 Compare

February 5, 2026 06:11

Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/large-4`	`77fae2`	`f8591d`
	`13987.92 MB/s`	`12148.77 MB/s`	`0.87`	`🔴`
	`1171.00 ns/op`	`1349.00 ns/op`	`1.15`	`🔴`
`FindSequence_Random/small-4`	`77fae2`	`f8591d`
	`5061.19 MB/s`	`4319.71 MB/s`	`0.85`	`🔴`
	`50.58 ns/op`	`59.26 ns/op`	`1.17`	`🔴`
`GenerateDocs-4`	`77fae2`	`f8591d`
	`949.50 ns/op`	`1106.00 ns/op`	`1.16`	`🔴`

Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/tiny-4`	`77fae2`	`ef9cdd`
	`2850.02 MB/s`	`2399.97 MB/s`	`0.84`	`🔴`
	`22.46 ns/op`	`26.67 ns/op`	`1.19`	`🔴`

forshev force-pushed the 290-filter-out-docs branch from 1e19ca1 to fc34615 Compare

February 5, 2026 14:33

Contributor

github-actions bot commented Feb 5, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/medium-4`	`77fae2`	`32ec58`
	`11769.73 MB/s`	`10062.12 MB/s`	`0.85`	`🔴`
	`87.00 ns/op`	`101.80 ns/op`	`1.17`	`🔴`
`FindSequence_Random/tiny-4`	`77fae2`	`32ec58`
	`2850.02 MB/s`	`2247.37 MB/s`	`0.79`	`🔴`
	`22.46 ns/op`	`28.48 ns/op`	`1.27`	`🔴`
`GenerateDocs-4`	`77fae2`	`32ec58`
	`949.50 ns/op`	`1120.00 ns/op`	`1.18`	`🔴`

forshev added 9 commits

February 6, 2026 13:45


          feat(config): allow multiple filtering requests

6ed1e2f


          feat(config): allow multiple filtering requests

ce2f9d7


          feat(config): allow multiple filtering requests

5f996e7


          feat: remove imitation PoC - write path

67eb078


          feat(docsfilter): store lids in binary format

050a1f7


          feat(docsfilter): store lids in binary format

0de713d


          feat(docsfilter): store lids in binary format

24a48eb


          feat(docsfilter): method to read filtered lids

e944da4


          feat(docsfilter): filter ids on search

2443add

forshev added 12 commits

February 6, 2026 13:45


          feat(docsfilter): filter ids on fetch for sealed fracs

b6dd334


          feat(docsfilter): support filtering on active fraction

97c10ff


          feat(docsfilter): refresh frac tobstones after sealing

dec9333


          fix: fix rebase

574653c


          fix: fix tests

c38162b


          chore(docsfilter): organize tombstone files in blocks

470caf5


          chore(docsfilter): compress lids blocks in tombstone files

dc4862f


          chore(docsfilter): handle errors

756c5fe


          chore(docsfilter): add iterator to lazy load tombstones


          chore(docsfilter): merge multiple iterators into one

a545350


          chore(docsfilter): use iterators in search and fetch

5098d47


          chore(docsfilter): remove tombstones when frac is evicted

ef49f29

forshev force-pushed the 290-filter-out-docs branch from fc34615 to ef49f29 Compare

February 6, 2026 08:47

Contributor

github-actions bot commented Feb 6, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`FindSequence_Random/medium-4`	`6ae568`	`070d19`
	`13517.73 MB/s`	`8992.43 MB/s`	`0.67`	`🔴`
	`75.75 ns/op`	`113.90 ns/op`	`1.50`	`🔴`
`FindSequence_Random/small-4`	`6ae568`	`070d19`
	`5683.27 MB/s`	`3800.59 MB/s`	`0.67`	`🔴`
	`45.04 ns/op`	`67.36 ns/op`	`1.50`	`🔴`
`GenerateDocs-4`	`6ae568`	`070d19`
	`957.90 ns/op`	`1118.00 ns/op`	`1.17`	`🔴`
`ProcessDocuments-4`	`6ae568`	`070d19`
	`4.00 allocs/op`	`5.00 allocs/op`	`1.25`	`🔴`


          refactor(docsfilter): move common parts from iterators

d455917

Contributor

github-actions bot commented Feb 6, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Bitmask-4`	`6ae568`	`ce7881`
	`0.00 B/op`	`3.00 B/op`	`NaN`	`🔴`
`FindSequence_Random/medium-4`	`6ae568`	`ce7881`
	`13517.73 MB/s`	`8591.23 MB/s`	`0.64`	`🔴`
	`75.75 ns/op`	`119.20 ns/op`	`1.57`	`🔴`


          chore(docsfilter): write metrics

e39db32

forshev force-pushed the 290-filter-out-docs branch from 2bfa582 to e39db32 Compare

February 6, 2026 13:57

Contributor

github-actions bot commented Feb 6, 2026

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Bitmask-4`	`6ae568`	`0023d4`
	`0.00 B/op`	`3.00 B/op`	`NaN`	`🔴`
`FindSequence_Random/small-4`	`6ae568`	`0023d4`
	`5683.27 MB/s`	`3844.97 MB/s`	`0.68`	`🔴`
	`45.04 ns/op`	`66.58 ns/op`	`1.48`	`🔴`

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet